High-throughput Publish/Subscribe on top of LSM-based Storage
نویسندگان
چکیده
State-of-the-art publish/subscribe systems are efficient when the subscriptions are relatively static – for instance, the set of followers in Twitter – or can fit in memory. However, now-a-days, many Big Data and IoT based applications follow a highly dynamic query paradigm, where both continuous queries and data entries are in the millions and can arrive and expire rapidly. In this paper we propose and compare several publish/subscribe storage architectures, based on the popular NoSQL Log-Structured Merge Tree (LSM) storage paradigm, to support high-throughput and highly dynamic publish/subscribe systems. Our framework naturally supports subscriptions on both historic and future streaming data, and generate instant notifications. We also extend our framework to efficiently support self-joining subscriptions, where streaming pub/sub records join with past pub/sub entries. Further, we show how hierarchical attributes, such as concept ontologies, can be efficiently supported; for example, a publication’s topic is “politics” whereas a subscription’s topic is “US politics.” We implemented and experimentally evaluated our methods on the popular LSM-based LevelDB system, using real datasets, for simple match and self-joining subscriptions on both flat and hierarchical attributes. Our results show that our approaches achieve significantly higher throughput compared to state-of-the-art baselines.
منابع مشابه
Work in Progress: PUB-2-SUB: A Content-based Publish/Subscribe Framework for Cooperative P2P Networks
This paper is focused on the content-based publish/subscribe service and our problem is to devise an efficient mechanism that enables this service in any given P2P network of cooperative nodes. Most techniques require some overlay structuralization added on top of the network. We propose a solution called PUB-2-SUB which works with any unstructured network topology. In addition, multiple indepe...
متن کاملTop-k/w publish/subscribe: A publish/subscribe model for continuous top-k processing over data streams
Continuous processing of top-k queries over data streams is a promising technique for alleviating the information overload problem as it distinguishes relevant from irrelevant data stream objects with respect to a given scoring function over time. Thus it enables filtering of irrelevant data objects and delivery of top-k objects relevant to user interests in real-time. We propose a solution for...
متن کاملEfficient matching for state-persistent publish/subscribe systems
Content-based publish/subscribe systems allow information dissemination and fine-grained information filtering in loosely coupled distributed systems. Stateless publish/subscribe systems send notifications to all subscribers whose subscriptions match an incoming publication. Statepersistent publish/subscribe systems, a recently proposed model that stores the states of both publications and subs...
متن کاملP3S: A Privacy Preserving Publish-Subscribe Middleware
This paper presents P3S, a publish-subscribe middleware designed to protect the privacy of subscriber interest and confidentiality of published content. P3S combines recent advances in cryptography, specifically Ciphertext Policy Attribute Based Encryption (CP-ABE) and Predicate Based Encryption (PBE) with an innovative architecture to achieve the desired level of privacy. An initial P3S protot...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2017